In the case of capturing an image by a digital camera, noise may sometimes be added to the image due to the characteristics of a CCD (Charge-Coupled Device) or a readout circuit for CMOS or the characteristics of transmission paths. Also, blur of an image due to an out-of-focus condition in capturing of the image or blur of an image due to camera shake occurs. In this way, the captured image has blur which is attributed to user's handling of the camera in photographing in addition to the noise which is attributed to the specific characteristics of the captured image, so that the image is degraded. Among such “blur” types, blur of an image which is attributed to a motion of a camera (camera shake) during photographing (exposure) is herein referred to as “motion blur”, so that it is distinguishable from blur that is attributed to an out-of-focus condition (out-of-focus blur).
In recent years, especially, demand for high sensitive photography is increasing, and therefore, it is necessary to restore an image degraded by blur (hereinafter, “degraded image”) to an image which is as close to an original image (hereinafter, “ideal image”) as possible. To realize a bright image which is free from noise or blur, such as an image demanded in high sensitive photography, the solutions are generally divided into two ideas, increasing the sensitivity and extending the exposure time.
However, increasing the sensitivity disadvantageously leads to amplification of noise. As a result, a signal is buried in the noise so that, in many cases, large part of a resultant image is formed by the noise. On the other hand, extending the exposure time enables accumulation of a larger amount of light which occurs at that site, resulting in an image which includes smaller noise. In this case, a signal would not be buried in the noise, but there is a problem of generation of motion blur in an image due to camera shake during the exposure time.
According to the prior art, there are two countermeasures against the problem resulting from the extended exposure time. One is optical camera shake compensation, such as lens shift, sensor shift, or the like. The optical camera shake compensation is disclosed in, for example, Patent Document 1. The other one is obtaining the direction/magnitude of motion blur from a resultant image and performing signal processing based on the obtained direction/magnitude of the blur to restore the image (a restoration method based on signal processing). The restoration method based on signal processing is, for example, disclosed in Patent Documents 2 to 4 and Non-patent Documents 1 to 7.
Extending the exposure time for the purpose of collecting a sufficient amount of light in a dark environment increases the probability of occurrence of camera shake. To avoid motion blur by means of optical camera shake compensation in such a dark environment, it is necessary to extend the operation range of a lens or a sensor. However, there is a problem that, when the operation range is increased, a time delay occurs in moving the lens or the sensor. Also, increasing the operation range encounters a physical limit.
A phenomenon that an image is degraded due to camera shake, from an ideal image to a degraded image, can be modeled as described below. A function which represents the brightness of each pixel of the degraded image is thought to be obtained by convolution of a function that represents the brightness of each pixel in the ideal image and a point spread function (PSF) that represents blur of an image. Restoration of the ideal image based on the obtained degraded image is realized by deconvolution of the degraded image and the PSF. A convolution operation is equivalent to a multiplication in the frequency space. Therefore, in the frequency space, the degraded image is divided by the PSF, whereby the restored image can be obtained.
Thus, when the PSF is known, the restored image can be obtained relatively readily by means of the above-described deconvolution so long as the effect of noise is neglected. On the other hand, when the PSF is unknown, it is necessary to estimate the PSF from the degraded image in order to obtain a restored image.
Estimation of the PSF may be realized by, for example, a method based on the sparse coding concept which is disclosed in Non-patent Document 1. According to this method, in the first place, a first restoration result is obtained from a manually-given initial PSF and a degraded image. Then, the first restoration result and the degraded image are used to estimate a PSF which is close to a true PSF. The initial PSF is amended with the estimated PSF. The amended PSF is used to obtain the second restoration result based on the degraded image. Subsequently, the step of obtaining the Nth restored image from the (N−1)th PSF and the degraded image and the step of estimating the Nth PSF from the Nth restored image and the degraded image are repeated, whereby a PSF estimation process and a restoration process performed on the degraded image are concurrently advanced.
However, this method has a problem that the restored image includes noise, such as ringing artifacts. “Ringing” refers to noise which makes a uniform brightness portion of an image (flat portion) appear uneven.
FIG. 1(a) is a plan view showing an image in which the brightness changes stepwise (an ideal image near an edge). FIG. 1(b) is a graph schematically showing the brightness distribution of the image. FIG. 2(a) is a plan view showing a degraded image (blurred image) obtained by photographing the image of FIG. 1(a) using a camera. FIG. 2(b) is a graph schematically showing the brightness distribution of the image. This is herein assumed to be a result of a horizontal sideward camera shake in photographing with a camera. The degraded image of FIG. 2(a) lacks sharpness at the edge due to camera shake blur (motion blur). FIG. 3(a) is a plan view showing an image restored by signal processing based on the degraded image of FIG. 2(a). FIG. 3(b) is a graph schematically showing the brightness distribution of the restored image. The restored image of FIG. 3(a) includes a portion in which the brightness periodically varies. Such a variation in brightness is noise called “ringing”. Ringing occurs due to the presence of a point in the frequency space at which the value of the PSF is zero (0).
The techniques of solving such a ringing problem are disclosed in Patent Documents 3 and 4 and Non-patent Documents 6 and 7. According to these techniques, light is blocked according to a predetermined time pattern (coded pattern) rather than keeping the shutter open during the exposure time, whereby a portion of the frequency space in which the PSF is zero (0) can be reduced. Therefore, occurrence of ringing noise can be prevented. The methods disclosed in Patent Documents 3 and 4 and Non-patent Documents 6 and 7 are called “Coded Exposure Photography”.